This invention generally relates to data processing systems, and more specifically, to such systems that simulate neural systems or networks.
Neural networks are designed to simulate, in a very simplified way, the operation of the human brain; and in a neural network, a multitude of input values are applied to a multitude of processing elements, referred to as neurons. For instance, a particular input value may be applied to one, several, or all of the neurons of the network; and a particular neuron may receive one, several, or all of the input values of the network as well as output values of other neurons in the network. Typically, each input value of each neuron is given a weight, and each neuron generates an output value that is a function of the sum of its weighted input values. The weight that is given to a particular input value by a particular neuron is determined by a data transfer function, which may be constant or which may vary over time.
A data processing system having just one processor, or a small number of processors, may be programmed to simulate the operation of a neural network, and in particular to do the same calculations and to generate the same output values as a neural network. More specifically, in a neural network, a multitude of sets of input values are conducted to a multitude of neurons, and each neuron processes a respective one set of input values according to a particular algorithm to generate a respective output value. In a data processing system designed to simulate a neural network, a multitude of sets of input values may be transmitted to just one processor but over a multitude of time periods; and the processor processes each set of input values over a respective one period of time according to a particular algorithm to generate an output value from each set of input values.
For example, in a neural network, a first set of input values may be applied to a first neuron, which processes these input values according to a first algorithm to produce a first output value; a second set of input values may be applied to a second neuron, which processes these input values according to a second algorithm to produce a second output value; and a third set of input values may be applied to a third neuron, which processes these input values according to a third algorithm to produce a third output value. In a data processing system designed to simulate the neural network, the first set of input values is transmitted to a processor over a first period of time, and the processor processes these input values according to the first algorithm to produce the first output value; the second set of input values is transmitted to that same processor but over a second period of time, and the processor processes these input values according to the second algorithm to produce the second output value; and the third set of input values is transmitted to the processor over a third period of time, and the processor processes these input values according to the third algorithm to produce the third output signal.
As a general rule, when the number of neurons in a neural network is increased, the network can handle more complex tasks and can better simulate human thinking. Thus, considerable attention has been given to designing neural networks having large numbers of neurons; and for example, neural networks have been designed with several thousand neurons.
A neural network having a large number of neurons may involve an extremely large number of calculations. Commonly, each input value of a network is applied to most or even all of the neurons of the network, so that calculating one complete series of neuron output values in a network having 200 input values and 200 neurons might require as many as 40,000 calculations. An additional 40,000 calculations might be needed, for just that one series of neuron output values, if the 200 neuron output values are fed back to all 200 neurons as additional input values. Further, with many neural network models, the output values are determined by differential equations whose solutions are a function of time. In these models, solutions are typically approximated by iteratively solving for the state of the network over discrete time intervals. Several iterations, and sometimes hundreds of iterations, are usually required to determine a stable solution for the network output values.
The processor or processors of any data processing system that would simulate a neural network having such a large number of input values and neurons would be required to perform an enormous number of calculations each time a series of neuron output values is generated. Even with high speed electronic data processing equipment, this huge number of calculations limits the speed at which the desired output values can be generated, and this limits the effectiveness with which the data processing system can simulate neuromorphic or neural networks.